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A two-part research and evaluation study was 
conducted to examine an elementary (K-8) school district's 
implementation of a computerized instructional management system, the 
"Computer-Managed Instruction/3000" (CMI), as a means of facilitating 
the assessment and updating of student mastery of 78 district-defined 
curricular objectives. The testing system consisted of a data bank of 
tests based on the district mathematics objectives, equipment to 
machine score student answer sheets, and reports detailing student 
mastery of the objectives. This system was piloted by administrators 
and teachers at four district-selected schools. A detailed interview 
schedule was developed to elicit such information as background in 
the district; knowledge of the CMI system, including how, why, and by 
whom it was selected; uses of the .system for testing and instruction; 
technical problems; potential uses of the system; and its effects on 
school organization and instruction. Analysis of the responses of 49 
-school— staff-members — principals7~res 0urce teachers, teachers, and 
clerical staff from both pilot and nonpilot schools — identified a 
number of factors as keys to the implementation of the program: (1) 
planning; (2) communication and awareness of its puroose; (3) support 
from key personnel; (4) training; (5) technical issues and 
procedures; and (6) rewards and incentives for using the system. When 
viewed positively, these factors operated as facilitators to the 
acceptance and implementation of the system; however, when viewed 
negatively, the same factors operated as barriers. Data summaries are 
provided in Tables 1-7. The Study Interview Schedule and the Coding 
Matrix are appended, and references are provided. (DJR) 
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Iapleoentatlon of a District Ccxrauterlzed Instructional 
Management Systea: Barriers and Facilitators 



This report presents results of a research and evaluation study 
conducted by the Far West Laboratory, in cooperation with an elementary 
(grades K-8) school district* The study examined the district 1 * 
Implementation of a computerized Instructional management system. The 
''Computer Managed Instruction/3000" system was selected by the district 
as a means to facilitate the assessment and updating of student mastery of 
78 district-defined currlcular objectives. The testing system consisted of 
a data bank of tests which were based on ti^e district mathematics 
objectives, equipment to machine score student answer sheets, and reports 
detailing student mastery of the objectives. The system was piloted by 
administrators and teachers at four district-selected schools. The 
objective of this paper Is to discuss the barriers and facilitators which 
were encountered during the pilot Implementation of the CMI testing system 
from January to June, 1985. 

This project emerged from a previous study conducted by the Instruc- 
tional Management Program at the Far West Laboratory for Educational 
Research and Development (FHLERD), a regional survey of Instructional 
management practices In school districts located In the Laboratory's tri- 
state region (Northern California, Nevada, and Utah). Educators reported 
computerized Instructional management systems as "exemplary" practices for 
Instructional Improvement (Rowan, Edelsteln, 4 Leal, 1985). However, the 
survey also elicited concerns about problems of Implementing district- 
level Instructional management systems. Examples of such concerns 
Included lack of training ot system users, lack of assessment, difficulty 
1n adapting systems to the needs of Instructional decisionmakers, and 
fears of evaluation by teachers (Rowan, 1984; Rowan et al., 1985). 



Perspectives 

Concrete guidelines for designing Instructional management systems have 
been provided by the literature on criterion-referenced testing (L1nn, 
Madaus, & Padulla, 1982), curriculum alignment (Mllazzo, Buchanan, & 
Schutz, 1981), mastery learning (Block, 1971; Bloom, 1976), and outcome- 
based education (Spady, 1982)1 However, less attention has been paid to 
problems and guidelines for Implementing such systems (Fllby, 1984). 

Studies of change and Implementation of inrtovatlons 1n schools and 
districts have shown that success of Implementation, Including district- 
wide administrative computing systems, depends upon many factors (Brown, 
1984). One Important factor 1s the support of the principal and other key 
personnel such as resource teachers (Brown, 1984). 

Roberts (1978, as reported by Brown, 1984) Identified potential 
facilitators and barriers to change; especially crucial to our analysis of 
the Implementation of the computerized Instructional management system are 
the key strategic factors of planning, support, and training, as well as 



the key process f&c tors of participation, communication, and motivation. 
This framework was useful 1n our analysis. 



CHI and District Background 

Oak Knoll Elementary School District (a pseudonym), a snedlum-slzed 
district with 18 elementary and 6 middle schools, Instituted a district 
curriculum based on CTBS-correlated performance objectives In 1983. The 
district adopted a plan requiring teachers to update their students 1 
progress periodically on the 78 district objectives. Teachers had been 
updating with a "raatrix" on which they recorded mastery, partial mastery 
or nonmastery for the objectives required for their grade level and 
subject(s). Some teachers had used their own tests for updating, while 
others used their judgment on their students 1 progress. The CMI system 
fit Into this larger district Instructional management program by provid- 
ing a systematic and standardized means for teachers to accomplish the 
task of updating. 

In 1984 the district purchased "Computer Managed Instruction/3000" 
(CMI), a software package for the HP 3000 computer. The CMI system was 
designed by a school district In another state. Using Its own objectives, 
Oak Knoll Elementary Schcol District (OKESD) constructed criterion- 
referenced tests (CRT's) correlated with CTBS objectives. The CMI can 
generate and machine score CRT's; update student records for mastery of 
district objectives while scoring student tests; generate reports on 
mastery of objectives with information on individual students, classes or 
grade levels within schools and across schools in the district; correlate 
objectives with Instructional materials and strategies; and allow teachers 
to input their own tests into a test data bank. 

The CMI pilot Implementation program was initially planned to start at 
the beginning of the 1984-85 academic year. However, this schedule was 
delayed due to problems in securing the necessary hardware. By the middle 
of the school year, hardware (consisting of a computer terminal, test 
scanner, modem, and phone lines) was Installed in four pilot schools 
designated by the district to Implement the testing system. While other 
schools had access to the tests, only some chose to use them with scanning 
equipment located at the district office. During the span of our study, 
the district utilized the CMI feature of district-developed CRT's 1n 
mathematics, and we thus focused on studying CMI use in mathematics. CRT's 
for reading and language arts were also being developed, but had only begun 
to be implemented by the end of the school year. 



Method 



Data Sources 

One source of Information was a Study Group formed of OKESD personnel. 
These district office and school staff (including principals, resource 
teachers/ teachers and clericals) met periodically with FHLERD project 
staff from January through October, 1985. The Study Group described the 
climate and history of the district, identified directions for Inquiry, 



and represented a variety of perspectives and attitudes within the 
district. 

A detailed Interview schedule was developed to elicit such Information 
as background in the district; knowledge of the CMI system regarding how, 
why, and by whom it was selected; uses of the system for testing and 
Instruction; technical problems; potential uses of the system; and effects 
of the system on school organization and Instruction (see Appendix). 
Responses to the Interviews served as the data sources for this study. 
[A second Interview schedule was devised for district office personnel to 
capture more detailed Information concerning CMI-reTated decision making, 
planning, coordination, and communication at the district level, but the 
analysis of those Interviews appears elsewhere (Crlst-Whl tzel, Edelsteln, 
4 Terry, 1985).] 



Saaple 

Sixty staff members from throughout the Oak Knoll District were 
Interviewed. Through a process of nomination by district and school site 
staff, 49 administrative and teaching personnel from four district- 
designated pilot schools and four similar schools not Implementing the CMI 
were selected for our school-level sample. There was an attempt to balance 
the sample with both users and nonusers, as well as to Include respondents 
with both positive and negative attitudes toward the system. Interviews 
lasting approximately one to two hours were held from March through June, 
1985 with 26 respondents from the four pilot sites (four principals, six 
resource teachers, 13 teachers, and three aides) and 23 respondents from 
nonpi lot schools (four principals, four resource teachers, and 15 
teachers). The sample is depicted in Table 1. [To obtain district-level 
perspectives for a separate analysis, 11 staff members from the district's 
central office were also selected for our sample.] 



Data Analysis 

Interviews were transcribed, and written summaries were prepared by the 
interviewers from audiotapes and handwritten notes taken during the 
Interviews. Analysis of the Interviews consisted of both qualitative and 
quantitative methods. A coding scheme, devised to correspond to the 
interview schedule and the range of responses to the interview questions, 
was used to generate frequency counts and to construct a general picture of 
response patterns that Illustrated various aspects of CMI use (see Appendix 
for the coding scheme). At the same time, a qualitative analysis of the 
interview protocols provided descriptive data including illustrative 
quotations and examples to provide more contextual detail. Both types of 
analysis were used in examining our data to Identify factors that seemed to 
operate as either barriers or facilitators to the CMI implementation. 



Table 1 
Interview Sample Description 
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Resource Teachers 


5 
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Teachers 


14 


16 


30 


Clerical/Aides 


3 




3 


Total for Schools 


26 


23 


49 


District Office Personnel 






11 


Total Sample 






60 



Results 



In examining the data gathered during Interviews with personnel In the 
Oak Knoll Elementary School District, a number of factors emerged as key to 
the Implementation of the Computer Managed Instruction/3000 program, which 
was designed to run on the district's HP3000. When viewed as positive by 
staff members, these factors operated as facilitators to the acceptance and 
Implementation of the CM I system. On the other hand, when viewed as 
negative aspects of the CM I by Oak Knoll personnel, these factors tended to 
serve as barriers to the successful Implementation of- the system. 

In this section we briefly discuss the major sources of resistance 
(barriers) and factors that facilitated the Implementation of the CMI, 
using the organizing themes of: (1) planning, (2) communication and 
awareness of Its purpose, (3) support from key personnel, (4) training, (5) 
technical Issues and procedures, and (6) rewards and Incentives for using 
the system. The sources of these data are the Interviews with 49 school 
personnel 1n the Oak Knoll School District Including principals, resource 
teachers, teachers and clericals from both pilot and nonpllot schools (see 
Table 1). This section focuses on overall responses from the four pilot 
and the four nonpllot schools. 



Planning 

Although the majority of both pilot and nonpllot personnel had 
positive attitudes toward the CMI (see Table 2), a number of personnel 
from all levels seemed to resist the CMI idea, some citing lack of school 
level Input as an explanation for their attitudes. Several Oak Knoll 
staff members who stated that the CMI plan was a "top-down 11 decision made 
by the board with minimal consultation from potential users, with Its use 
mandated at the school level. Further 1t was the opinion of 10% of the 
respondents (see Table 3) that there was no Input from any teachers or tne 
teachers 1 association In the last-minute decision to use the CMI system 
designed by another school district. One staff member claimed tne primary 
concern of the CMI Implementation was the needs ot the district 
administration; the needs of the classroom teacher were not- appropriately 
considered. A number of staff believed they had Ideas that could be 
helpful in the development, Implementation and effective use of the 
system. For example, an administrator mentioned that the expertise of 
mentor teachers could have been useful in developing the CMI tests. 
Teachers were later instrumental In developing CMI tests for reading. 

Poor planning was another frequently-mentioned CMI problem. The late 
Implementation of the CMI was cited as an example of this. It was 
reported that CMI service, which was to have begun in the fall, did not 
actually begin until the spring. Phone lines and computer hardware were 
not 1n place until that time. Interviewees (20%) felt that this was too 
late 1n the year for implementation (see Table 3) as 1t interfered with 
other district and classroom activities, such as spring CTBS testing. 



Table 2 
Attitude Towards the CMI 
Indicated by Pilot and Nonpilot Respondents 



Percentage of Respondents 

(n=26) (n«23) (N«49) 

Pilot Nonpilot Total 

Positive 58 52 55 

Negative 12 9 10 

Uncertain 27 35 31 

Unknown 3 4 4 
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Table 3 
Sources of Resistance 
Indicated by Pilot and Nonpilot Respondents 



Percentage of Respondents 



(n=25) (n=23) (N-49) 
Pilot Nonpilot Total 
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Lack of support from principal 


1 9 




12 
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Lack of support from district 


1 9 


22 

Cm Cm 


16 


Processing/reporting too slow 
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26 
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Problems with reports/data 


31 


1 7 


99 
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Fear of evaluation 


16 
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Inadequate training 


35 


22 


28 


Too much paperwork 


15 


30 


22 


Too much class time 


19 


17 


18 


A lot/ too much time 


31 


- 26 


28 


Coercion - mandated by district 


8 


9 


8 


Lack of confidence in tests 


23 


9 


16 


Teachers not asked for input 


12 


9 


10 


Coordination and supply of materials 


19 


13 


16 


Other 


31 


34 


33 
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This response was more evident among pilot school respondents (272) than 
nonpllot respondents (13%), as they were the group with the heaviest 
Involvement In Implementation. 

Another problem cited was "the lack of clear-cut short-term and long- 
term goals," which resulted In poor communication and contradictory 
Information being provided to schools. Some district personnel (28%) felt 
that too many new programs had been Introduced In a comparatively short 
period of time. Nonpllot respondents were somewhat more Inclined to 
propose such a reservation about the Implementation of the CMI than pilot 
respondents (34% versus 24%). Along with CMI last year, there were new 
programs In reading, language, social studies and spelling; there were 
school plans to be Implemented, and state-mandated evaluation of schools. 
Due to other academic demands, a number of teachers were reluctant to 
embrace new commitments. One teacher said, M I don't have the time to 
Incorporate new tests Into my Instruction. 11 



Coanunl cations and Awareness 

To develop awareness of the CMI, clear communication to staff at the 
school level is needed. Widespread knowledge about the CMI and Its 
purpose should facilitate Its acceptance and use In schools. 

We found that the existence and purpose of the CMI system were not 
well known In the district. In fact, several Study Group members and 
staff members being Interviewed asked for Information about the CMI 
because they had not been aware of Its existence and/or Its purpose. 

We found that perceptions of the purpose of the CMI system varied from 
Instructional to non-Instructional uses. The Instructional uses that were 
mentioned Included the role of the system In focusing the content of 
Instruction on the terminal objectives. The non-instructional uses cited 
were monitoring student progress and updating mastery; assessment of 
mastery and testing as related to the CMI; evaluating teacher performance; 
Increasing test scores; standardizing district measures of mastery. 

While some Interviewees mentioned only one purpose, others mentioned 
several of the six general categories of purposes (see Table 4). The rate 
at which each purpose was mentioned tended to be consistent across both 
pilot and nonpllot schools: 

o monitoring student progress Including the updating of the 
mastery of terminal objectives (51%) 

o testing students in order to assess their mastery of the 
curriculum objectives, and how this correlates with the CTBS 
test (51%) 

o standardizing the district measures of mastery (41%) 

o focusing Instruction and the content of the curriculum on the 
78 terminal objectives (33%) 

o evaluating teachers and their performance (10%) 



Table 4 
Purposes of the CMI 
Reported by Pilot and Nonpilot Respondents 



Percentage of Respondents 





(n=26) 


(n-23) 


(N=49) 




Pilot 


Nonpilot 


Total 


Monitoring 


46 


57 


51 


Curriculum 


31 


35 


33 


Testing 


58 


43 


51 


Evaluation 


12 


9 


10 


Increasing Test Scores 


8 . 


4 


6 


Standardization 


38 


43 


41 



o Increasing student test scores on the CTBS through practice 
with CMI tests (62) 

The perceived purposes of the CMI can serve as either barriers or 
facilitators to Its Implementation. Twenty-four percent of the school 
staff who were Interviewed perceived the CMI testing system as an 
evaluative tool, which was a source of resistance for some (see Table 3). 
In nonpllot schools, 352 of the personnel Interviewed felt that the CMI 
had the negative potential of being used to evaluate teachers, a 
perception of Its purpose which thus posed a barrier. On the positive 
side, principals and teachers alike reported that frequency of testing and 
results from updating could be used as a measure of effective teaching. 
This perception could be a facilitator to the Implementation process. 
When looking at these figures for teachers only, we observed that some 
teachers perceived the use of the CMI testing system for administrative 
purposes as a positive measure or facilitator (202), while other teachers 
(262) were reluctant to use the system due because of Its potential use in 
evaluating teachers. 



Support from Key Personnel 

One factor mentioned by Oak Knoll staff concerning the acceptance of 
the CMI was support from key personnel In the district. Lack of support 
from key personnel was a source of resistance. The school staff we 
Interviewed mentioned lack of support from a variety of sources (see Table 
3): the district office (162), principals (122), respected teachers (42), 
and the teachers 1 association (42). Some principals reportedly showed 
little Interest In the system and didn't feel Its use was necessary. As 
one teacher said about a principal's policy toward the CMI, "It was left 
open as to use. 11 

It was suggested by one principal and three pilot teachers that the 
district could show support for use of the CMI testing system by providing 
materials, for example, duplication of the CMI tests rather than having 
the schools or teachers absorb the costs. 



Training 

Training provides an additional form of support that can facilitate 
Implementation of an Innovation. Lack of training, on the other hand, may 
be an Inhibitor to the Implementation process. Teachers at schools using 
the CMI testing system received a workshop In testing administration and 
scoring procedures. Some school sites had scanning equipment (a test 
scanner, a computer terminal connected by modem to the district's HP3000). 
At these sites, students' answer sheets were run through the scanner to be 
scored. Many teachers did their own scanning. At other sites, students 1 
answer sheets were sent to the district office for scanning and scoring. 

While Inadequate CMI training was cited as a problem by 202 of the 
school staff respondents (see Table 3), teachers who didn't u*e the 
scanning equipment considered their training to be satisfactory for using 
the CMI testing system. When respondents cited problems with training, 
they referred mostly to the absence of training In using the CMI for 
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Instructional purposes, e.g., interpretation of test results for 
diagnosing student difficulties and prescribing appropriate instruction. 
Twenty-four percent 242 of the school staff recommended more training for 
instructional uses (see Table 5). The need for inservicing on how to read 
and Interpret the reports was mentioned in addition to how to use teacher 
made tests on the system. 

Regarding technical training on the CMI, while raany interviewees 
reported It to bs generally satisfactory and helpful, several people said 
they needed more *hands-on, experientlally oriented" training. While this 
"hands on" experience was cited as the best part of the training, some Oak 
Knoll staff felt they needed more time using the equipment In a practice 
situation. The need for such "hands on" training, of course, varies for 
different Individuals. 

Perceptions of the technical training varied by school and by how much 
interaction the teachers were required to have with trie scanning 
equipment. Some Interviewees mentioned that too much information was 
covered in one session. One respondent rated the training as helpful but 
insufficient for teachers to be able to use the scanner alone. More 
technical training was recommended by 162 of the staff Interviewed (see 
Table 5), 



Technical Issues and Procedures 

In the Implementation of any new program or Innovation dealing with 
technical equipment (such as the computers and scanners needed for the 
CMI), any problems related to the equipment and its operation will 
undoubtedly affect the attempts to use the program. As noted 1n Table 
3, technical problems were reported as the most significant source of 
resistance to the CMI (392), especially by the pilot school staff (652) 
who were actually exposed to the CMI equipment. 

Technical problens. Numerous problems with the CMI contributed to a 
negative Image of its effectiveness according to a number of school staff 
in the Oak Knoll district. As reported In Table 6, these problems 
included hardware (432), test format (352), software (312), test quality 
(292), technical output (222) and system compatibility (62). 

Difficulty with the scanning equipment was a major harder* problem. 
The equipment has been described as slow, variable in Its reliability, 
and "oversensitive to heat." Pre-slugging did 
not always work and answer sheets had to.be rebubbled 1n a number of 
cases. One source used a variety of procedures to scan 
pre-slugged answer sheets after a first attempt had failed: rebubbling 
the answer key, rebubbling student information, rebubbling student 
answers, each procedure followed by attempting to scan the answer sheets 
again. Additional procedures Included rebooting the scanner, then 
reprogramming it before trying to scan the answer sheets again. A last- 
resort measure was standing at the scanner with one's hand on It. A 
FWLERD researcher observed the success of this seemingly Innocuous measure 
after numerous attempts to can a set of answer sheets had failed. 
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Table 5 

Recommendations for Future Use of the CMI 
Indicated by Pilot and Nonpilot Respondents 



Percentage of Respondents 



Pilot (n=25) Nonpilot (n=23) Total (M*49) 



New, clearer test copies 15 9 12 
One key person responsible 

for scanning on site 31 0 16 

More technical training 19 13 16 

31 17 24 



Training for instructional 
uses 



More administrative support 

in building and district 23 22 22 

Other 19 35 27 
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Table 6 
Technical Problems 
Reported by Pilot and Nonpilot Respondents 



Percentage of Respondents 

(n=26) (n=23) (M=49) 

Pilot Nonpilot Total 



Hardware 73 9 43 

Software 42 17 31 

Technical Ouput 23 22 22 

System Compatibility 8 46 

Test Format 35 35 35 

Test Quality 35 22 29 

Other Problems 12 4 8 
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The test format was cited as a problem. One person noted that 1t was 
unlike the CTBS format, questioning whether this format actually assisted 
or confused students. Others complained about the graphics, the print 
clarity and the layout. There were also problems with the use of answer 
sheets for young elementary students, particularly for those In 
kindergarten. 

School staff detected one particular software problem early In the 
Implementation process: student answer sheets, allowing 100 test Items, 
could only be used for one test (usually 8 to 10 Items). Answer sheets 
were expensive. In addition, since all Information on the answer sheet 
was required to be completed before scoring, It was time-consuming to use 
a different answer sheet for each test. The district staff working on the 
CMI Implementation later solved this problem. Teachers were then able to 
use one answer sheet for multiple tests. 

Reportedly there was variation In the quality of the tests. Certain 
tests did not cover the full range of skills In particular objectives, for 
example, eight Items on measurement were not sufficient to test mastery. 
Other tests were reportedly too easy. For example, one teacher cited a 
geometry test on which all of the students In a particular class 
unexpectedly earned 100%. Some reading tests reportedly contained 
vocabulary that was too difficult for the students at that level. 

Technical output was also a problem. For a number of schools, the 
sorting procedure was Inadequate. For middle schools, the matrix and 
scanner answer sheets only came grouped by core class and had to be 
resorted by hand Into subject sections. This reportedly required several 
hours to complete. Sorting was also a problem 1n elementary schools where 
blank student answer sheets were grouped by grade level. It was also 
reported that processing was very slow, resulting In very slow turn-around 
time. According to a few sources, some test results were never returned 
to the teachers. One user of the system estimated that it took 
approximately 25 minutes to scan answer sheets for one class of 30 
students. A clerical aide at a middle school estimated that she spent two 
to three hours per day handling tests for the four matn teachers who used 
the CMI. In one Instance, It required two days to scan the multiple tests 
one middle-school teacher gave to her five math classes. 

Finally, there was some concern about the CMI's compatibility With 
other systems. For example, teachers questioned whether computerized 
report cards and reading tests currently scored on non-CMI scanning 
equipment could be used with CMI scanning equipment. 

Procedures and test adnlnl strati on. Many respondents reported that 
teachers spent a lot of time on tasks related to using the CMI: 7% of the 
teachers Interviewed and 142 of all school respondents spent 16 or more 
hours a month. Preparatory tasks Included copying exam questions; 
searching through files for exams; sorting answer sheets; bubbling answer 
keys and student Information on answer sheets. During class, time was 
occupied with explaining procedures; handing out pre-slugged answer sheets 
to the appropriate students; passing out exams; administering the exams; 
and collecting answer sheets and exam sheets. After the exam, time was 
spent resorting answer sheets with the appropriate keys; hand scoring 
exams or sending/ taking them to be scored; and scanning answer sheets. 
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Once the test results were returned from the district office, tastes 
Included Interpreting test results for grouping, planning and diagnostics; 
recordkeeping on student mastery; filing reports and test papers; and 
re teaching when appropriate. Additional time was spent making sure all 
students had number 2 pencils; securing enough answer sheets; and waiting 
for test results to return. 

Some teachers who personally scanned their students 1 tests had 
problems gaining access to the CMI system at their particular scnools. 
Without a free period or prep time, teachers reported that they. had to 
resort to using the system before school, during lunch, or after school, 
yet their contractual arrangements required them to be at school only 20 
minutes prior to the beginning of school and to stay only 15 minutes after 
school. Thus, teachers found It difficult to scan their own tests within 
the limits of the contractual teaching day. 

Lack of time also Inhibited teachers 1 attempts to use the system to 
enhance their teaching. One purported benefit was to Improve teachers 1 
self-evaluation and consequently their Instruction. However, teachers 
responsible for scanning their students 1 tests spent much of their time 
dealing with the equipment and its frustrations. There was little time or 
initiative left to plan lessons around the test results. 

While full sets of answer sheets could be run easily on the scanner, 
scoring answer sheets for just a few students was cumbersome. Each test 
required a separate answer key. Therefore, when several students took 
several different tests, multiple answer keys were required for each 
class. A number of teachers were hand-scoring CMI tests at the end of the 
school year in order to get immediate feedback on minimum proficiencies 
for a few students at a time. 



Incentives 

The benefits and motivations mentioned by respondents for using the 
CMI are shown in Table 7. Classroom personnel used the CMI system for 
testing and updating primarily because it was mandated (according to 
interviews with 37% of all school staff and 42% of pilot school staff). 
The CMI system was also used because of its potential benefits; Its ease 
and speed in testing and updating students on the curriculum objectives; 
and from curiosity or interest In how the system worked. It was also said 
that some classroom staff used the CMI system to please or appease their 
principals. ,~ . 

Twenty-two percent of the school staff interviewed were motivated to 
use the CMI for Its potential benefits and uses, both instructional and 
noninstructional. The Instructional uses Included information for 
grouping, planning, pacing, and diagnostics. Noninstructional uses were 
assessment of class and teacher progress for administrative uses, parent 
conferences and student conferences. Potential benefits served as an 
incentive more often for nonpilot respondents (30%) than for those 1n pilot 
schools (15%), perhaps due to the fact that use in nonpilot schools was 
voluntary. We found that teachers who had positive attitudes about the use 
of the system also perceived more current and potential uses of the system 
which would benefit them. Those teachers who had negative or uncertain 
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Table 7 

Motivations and Incentives for Using the CMI 
Indicated by "Pilot and Nonpilot Respondents 



Percentage of Respondents 



Pilot Nonpilot Total 

(n=26) (n=23) (N=49) 



To please/appease the principal 


8 


13 


10 


Mandated 


42 


30 


37 


Curious/Interested 


8 


13 


10 


Potential Benefits 


15 


30 


22 


Easier and/or faster 


19 


22 


20 
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attitudes toward the system tended to perceive fewer Instructional purposes 
and applications for the system, and little benefit for themselves. These 
teachers also did not use the system as widely. 

Several veteran teachers we Interviewed felt they could have 
contributed considerably to the development of a successful program. They 
additionally felt that their skill and expertise were not being 
acknowledged at the district level, resulting 1n negative feelings and 
suspicion about the validity of the tests by some school staff {16% as 
shown In Table 3.) On the other hand, several teachers who reported that 
their expertise had been employed In developing reading tests seemed 
enthusiastic and positive about the CMI system, Its validity and Its use. 
This example demonstrates how positive attitudes were generated towards 
the CMI by soliciting teacher Input. 

Several respondents reported that the CMI system Itself carried no 
rewards or Incentives. Therefore, school staff with this negative 
attitude did not see any advantage for themselves In using the system, 
only additional work for those who were already overloaded. One source 
called the system "a new toy" for the district to play with, which was 
being Imposed on teachers. 

The perceived purposes of the system can serve as positive or negative 
Incentives for using the CMI. Negative perceptions of the CMI's purpose 
were deterents to Its use. A number of teachers felt the purpose of the 
system to be reporting and standardization, and without Instructional 
application. This resulted In some teachers testing with the CMI without 
looking at the reports for Instructional guidance or seeing any value in 
them. 



Discussion 

The majority of all levels of personnel Interviewed had positive 
attitudes toward the CMI (wltn a relatively small percentage being 
outright negative), which should serve as a potential general facilitator 
for further Implementation efforts. However, 1n spite of such a tendency 
toward a receptive overall disposition, Interviewees reported a number of 
factors which operated as barriers to implementation. Some of the major, 
barriers to use of the CMI by school personnel were: a perception of lack 
of planning by the district; lack of knowledge about the CMI; lack of both 
communication and consensus about Its purposes, potential uses, and 
benefits; lack of support from key personnel at the district and school 
levels; lack of training, especially for Instructional uses; a range of 
technical problems encountered with both the system hardware and software; 
lack of time to devote to the tasks accompanying the system's use; and 
lack of perceived Incentives or rewards for using the system. Especially 
troublesome for many teachers trying to use the system were the numerous 
technical problems, such as operating the scanner, dealing with answer 
sheets, and sorting reports. 

School staff suggested various factors, on the other hand, which 
might facilitate the CMI Implementation and use by teachers: using the 
skill and expertise of teachers In the planning process and 1n test 
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development; knowledge and understanding of CMI purposes, benefits and 
potential uses; support for Implementation from principals and the 
district office; the technical support provided by the district and by 
site technical personnel; and the provision of training to use the CMI 
Instructlonally. The presence or absence of a particular factor may 
operate either as a barrier or facilitator to the Implementation. The key 
seems to be to capitalize on the "state" of each factor that 1s 
facllitatlve rather than the "state" which Inhibits the CMI's use. 



Putting the Findings 1n Perspective 

It Is Important to point out that effecting an Innovation takes 
considerable time. Our study encompassed only the first half-year's 
effort at a pilot Implementation: its problems and successes need to be 
considered In that light. Although OKESD began Its CMI Implementation 
several months later than originally planned, such a late start Is 
probably not unusual given the technical nature of the CMI hardware and 
software and the problems that occurred as a function of the complex 
technology. In another study of the Implementation of computerized 
management systems In school districts, 1t was found that It took the 
districts five to eight years to evolve their programs to a relatively 
advanced state (Williams & Bank, 1984). 

In view of the relatively short time span we studied, It Is important 
to acknowledge the successes of the CMI pilot effort. First of all, It 
was Intended to be a pilot "test": any pilot test Is going to surface 
problems and "bugs". To the extent that the district Surfaced" all the 
technical "bugs" that It did and solved many of them (Including many of 
those we have listed In our report), the pilot test can be considered a 
success, albeit perhaps a qualified one. Second, the district learned a 
good deal about the technical nature of the system and about Its use in 
selected schools. The knowledge the district gained about the factors 
which encouraged Implementation, as well as those which hindered use of 
the CMI, provides Information that can be used constructively In planning 
and coordinating the expansion of the CMI Into additional currlcular areas 
(such as reading and language) and more school sites. 



Theoretical Perspectives on Change 

It seems appropriate here to discuss our findings within the readily 
applicable theoretical framework outlined by Roberts (1978, as discussed 
by Brown, 1984). A number of factors derived from the Interviews with 
both pilot and nonpilot school personnel, as well as district staff, 
were determined to be facilitators or barriers to the implementation of 
the CMI during Its pilot phase 1n the Oak Knoll School District. The 
notions of barriers and facilitators are key concepts for OKESD, as for 
any organization attempting an Innovation, to take account of 1n the 
process of implementing the CMI system throughout the district. 

Roberts delineated critical potential barriers and facilitators 
relevant to the introduction of computers into schools for administrative 
uses; a framework which seems equally appropriate to consider in the 
present context of Introducing computers Into the district and schools for 
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Instructional management purposes. 



Roberts 9 potential barriers. Roberts' first barrier Is the 
complexity of the computer Itself 1n terms of both hardware and software, 
which makes - Its acceptance problematic. This was evidenced In the variety 
and magnitude of the technical problems encountered 1n Oak Knoll's CM I 
Implementation effort. 

Roberts' second proposed barrier entails the resources required , 
Including ample time and effort needed by personnel to use the computer as 
Intended. In Oak Knoll, this barrier was evidenced by lack of time and 
access to equipment by teachers; In some cases, lack of adequate personnel 
to perform the technical functions; and lack of material resources such as 
test copies and answer sheets. 

Third, Roberts points out the demand for understanding as a barrier 
1n that the "mysterious" nature of the computer to many staff members 
Inhibits their use of It. This barrier can translate to the Inability of 
some Oak Knoll staff to operate the equipment effectively; their 
descriptions of the scanner as being "sensitive" or requiring the "magic 
touch" to make It work; their hesitancy about even getting their "hands 
on" the computer; and their reported need for more training. 

The fourth barrier suggested by Roberts— the need for role change or 
alterations In ways personnel perform administrative tasks— operated as a 
barrier in some cases where schools did not clearly delineate such new 
roles or struggled with a need for them (as In the case of the school 
which eventually turned to designating a specific staff member to perform 
the necessary technical functions of the CM I system). 

Roberts 1 change facilitators. Roberts also outlined a number of change 
facilitators that apply directly to the Oak Knoll CMI Implementation 
effort. Some facilitators are classified as strategic principles : (1) 
planning oriented toward the computer user and allowing sufficient 
preparation time; (2) support, Indicating high-level administrative 
endorsement and commitment of resources; and (3) training that 1s adequate 
and that allows for Individual differences among users. 

Other facilitators are classified by Roberts as process factors : (1) 
participation that Includes shared decisionmaking and involvement of all" 
users; U) communication that Includes commonly understood terminology and 
strategies for reducing potential conflicts; and (3) motivation, including 
opportunities for professional growth and recognition of accomplishments. 

We have directly discussed all of these barriers and facilitating 
factors under related headings and we have pointed out that the apparent 
lack of the change facilitators has, 1n turn, posed barriers in various 
ways to the district's attempts at implementing the CMI, such as those 
barriers above posed by Roberts. 



Significance 

Many school districts are focusing on outcome-based education, mastery 
of basic skills by students, standardized currlcular objectives, alignment 



19 

21 



of testing to curriculum* and technological Innovations as efforts at 
school Improvement* Thus, Interest 1n computerized Instructional 
management systems will Increase. Beyond the usefulness of this study to 
the district Involved^ other districts planning or beginning to implement 
such a system can benefit from Oak Knoll's experience, Its problems, and 
Its successes. Knowledge of the factors that encouraged and discouraged 
successful Implementation of the CMI can serve as a guide to their own 
efforts to use successfully a computerized Instructional management system. 
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STUDY INTERVIEW SCHEDULE 

Personal History 

1. Tell me about your history In the district . 

Uses of the CM I 

2. (Purpose/Communications) 

Tell me what you know about the district's computerized Instructional 
management system . 

How and when did you learn about the system? 

What was your Initial reaction to the CHI? How f.is that changed? 

3. (Instruction) 

How Us Instruction been affected by the CHI (or updating)? 

(Each content area separately: math, reading, language arts) 
(grouping, pacing, diagnostics, planning, etc.) 

4. (Curriculum/Content) 

How has the content of Instruction been affected (In the 3 updating areas)? 
( Non-users : by the updating of the terminal objectives) 

( In math - 1n reading - 1n language arts? ) 

Which text do you use? 
Do you use the text differently now? 

Does the CHI (or updating) affect the scope and sequence of the curriculum? 
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(Testing) 

Do you do more or less testing with the CHI (or updating) system? 
How often do you test now? 
How did you test before the Cftl? 

What Information can you obtain frora the CMI or updating system? 

How do you use the test results? 

How useful are the test results for your teaching? 

CHI users ; Is testing easier or harder using CMI? 

(Updating Activities) 

Do you update student objectives? 

What kind of system do you presently use to update student progress on the 
district objectlve"sT 

In oath, reading, language arts 

How much time do you ti on activities or tasks related to the system? 
dally, weekly, monthly - 
H^-.w do you spend that time? 
Do you spend more or less time than before? 

What did you use before your present system? 

27 



7. (school/Work Organization) 

Do you organize your classes differently as a result of the CHI and updating? 



Has there been any change In the way you work with other staff members In 
your school? 

for coordination, planning, grouping 

1n the division of labor 



8. (Evaluation) 

What do you like and dislike about the system? 



9. (Usage: Potential) 

What are the potential uses of CHI? 
for teaching 

for teachers' relations with parents/or students 
for administration 
for the district 
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Technical Uses of the CMI 

1. How many terminals or scanners for use with the CHI system are located 
at your site? 

Where 1s the equipment located? 

When can they be used? 

Who 1s 1n charge of the equipment? 

Who are the primary users? (name" them and their job titles) 

2r Kave you experienced any technical problems with the system? 
ftas there been any "down time?" 

3. Is the Information from the computer output presented 1n a clear and usable 
form? (matrices, tests , CMI report;) 

Understandable to teachers? 

parents? 
students? 
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Training 

1. Have you received any CMI training? 

What did it consist of? 

Who gave the training and how long was it? 

What was effective and what wasn't? 

Is there ongoing training and support? 
If yes : tell me more about it. 



2. What means are available to you to voice opinions regarding the system 
and Its performance?" 
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CODING MATRIX 
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Personal History 
1 



00 Other 



Highest Grade 
0 Hon- teacher 
K - 8 

9 Unknown 



2 Personal Identification number 

10 George 

20 Hubbard 

30 Ocala 

40 Meyer 

50 Sheppard 

60 Shields 

70 Slonaker 

80 Pal a 

90 District 

3-4 Position/Grade 

T Teacher 

R Resource Teacher 

P Principal 

A Aide 

C Clerical 

E Comp Ed 

V Evaluation 

U Curriculum 

5-6 Years 1n district 
99 Unknown 

7 Pilot status of school 1n which employed 
P Pilot 

N Nonpllot 

8 Level of school In which employed 
E Elementary 

M Middle 

9 Card #1 or 2 
10-11 Blank 

Purpose/Communications (statement of purpose, use of system) 

12 M Monitoring student progress - updating 

13 C Curriculum - focus on content end instruction/objectives 

14 T Testing - CTBS and assessment of aastery 

15 E Evaluation of teachers, teacher performance 

16 S Increase Test Scores 

17 D Standardization of district measures of mastery 
18-19 Blank 



\ 



CODING MATRIX p. 2 

Curriculum/ Con teii t 

20 Content Focus: Choose the one that applies. 

0 On objectives (objective-driven Instruction) 

T On text or other 

X Unknown, not mentioned 

M On minimum proficiencies 



21 Change 1n textbook use 

Y Yes, there has been a change 
N No, there hasn't been a change 
X Not mentioned, unknown 

22 Change 1n scope of curriculum 
M More extensive 

L Less breadth 

N No change 

U Coder uncertain, protocol unclear, respondent undecided 

X Not mentioned, unknown 



23-25 Blank 



Testing 

26 Change 1n amount of testing with CHI/updat1ng 
M More testing 

L Less testing 

'N No change 1n amount of testing 

U Uncertain, unclear, undecided 

X Not mentioned, unknown 

27 Difficulty of testing with CMI/updat1ng 
M More difficult to test 

L Less difficult to test 

N No change 1n testing 

U Uncertain, unclear, undecided 

X Not mentioned, unknown 

C Conditional 

28 Type of tests used 
C CMI 

U Unit tests from texts 
0 Teacher's own tests 
X Unknown 

F Unit tests with teacher's own tests 

G CMI with unit tests 

H CMI with teacher's own tests 

J CMI with unit and own tests 

» 

29 Whether student performance on the CTBS objectives affects grading 
Y Yes 

N NO 

X Unknown, uncertain, unclear 



COOING MATRIX 

Instruction applications - test results and/or matrix: Non-CMI 



30 - 


37 


How test Information Is used: code each of the 
following that apply; put In X when It doesn't. 


30 


6 


Grouping/Placement 


31 


L 


Planning 


32 


D 


Diagnostics for Individuals 


33 


P 


Pacing 


34 


M 


Mastery assessment 


35 


C 


Minimum competency for promotion/graduation 



Other applications of test results and/or matrix: Non-CMI 



36 


Z 


Parent conferences 


37 


S 


Student conferences 


38 


F 


Family tutoring 


39 


E 


Evaluation 


40 




Blank 



Instruction 



41 - 


51 


How CMI Information Is used: code each of the 
following that applies; put In X when It doesn't. 


41 


G 


Grouping 


42 


L 


Planning 


43 


D 


Diagnostics for Individuals 


44 


P 


Pacing 


45 


M 


Mastery assessment 


46 


C 


Minimum proficiency/competency 


47 


Z 


Parent conferences 


48 


S 


Student conferences 


49 


F 


Family tutoring 


50 


E 


Evaluation 


51 - 


52 


Blank 
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CODING MATRIX 



Updating 

53 Updating other areas 
" . C CMI 

M Matrix 

N Nothing 

Z Subject area not applicable 

X Unknown 

54 Present system used by person for updating math only 
C CMI 

M Matrix 

N Nothing 

Z Subject area not applicable 

X Unknown 



55 - 56 Blank 
Time Use 

57 Time spent on CMI , ^dating, etc. (1n hours per aonth: 
multiply dally by 20, weekly by 4) 

1 0-5 hours/month 

2 6 - 10 

3 11-15 

4 16+ 

9 Unknown 



58 Change 1n time use 
M More 

L Less 

N No change 

U Uncertain (interviewee) 

X Unknown (to coder) 

C Conditional 

59 - 64 Tasks 

59 P Planning 

60 S Scanning 

61 R Recordkeeping and clerical 

62 A Scoring 

63 M Meetings 

64 0 • Other 

65 Blank 
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CODING MATRIX 



Card 2 

Potential Usage 



12 - 


24 


Possible uses for the system: code each of the 
followlna that aoolv: out 1n X when 1t doesn't. 


12 


u 


Uodatl no/Placement 

^F BF OA Will ^B § 1 ■ ^F BF IMF II BP 


13 


G 


Grouolna 


14 


1 

A* 


Lesson Plannlna 


15 


D 


Diagnostics for Individuals 


16 


p 


Pacing 


17 


M 


Mastery Assessment 


18 


c 


Minimum proficiency/competency 


19 


z 


Parent conferences 


20 


s 


Student conferences 


21 


F 


Family tutoring 


22 


E 


Teacher evaluation 


23 


T 


Testing 


24 


0 


Other 


25 - 


26 


Blank 



Technical 

27 Primary users of hardware 

T Teacher 

P Principal 

R Resource teachers 

C Clericals 

0 Other 

A Aide 

X Unknown 

N Nonpilot 
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CODING MATRIX 

p. 6 

Technical (continued) 

28 Location of equipment 
T Teacher workroom 
L Library 

0 Office 

P Principal's office 

Z Other 

X Unknown 

N Nonpllot 

29 - 34 Technical Problems: code each that applies; put X when 

It doesn't. 

29 H Hardware 

30 S Software 

31 0 Technical Output (e.g Inaccuracies , missing Info) 

32 C System compatibility (Incorporating other systems Into CMI) 

33 F Test format (Including legibility) 

34 Q Test quality (too hard, too easy, too short, proper alignment) 

35 U Problems: Source unknown 

36 - 37 Blank 

Report Use : Is the output OK? Use the following codes: 

38-41 H Helpful 

S Satisfactory 

N Some problems 

M Many problems 

X Unknown, no comment 

38 For teachers: format 

39 For teachers: content * Information 

40 For parents 

41 For students 

42 - 43 Blank 

35 
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CODING MATRIX 

P. 7 

Training 

44 Has tralnlng/lnservlce received? 

Y Yes 
N No 

X Unknown 

45 Was the training effective? 
H Helpful 

S Satisfactory 
N Negative aspects 
X Unknown 

46 Is there ongoing support? 

H Ongoing support: Helpful and of high quality 

S Ongoing support: Satisfactory 

N Not received or negative aspects 

X Unknown 

47 - 48 Blank 
Administrative Issues 

49 Was respondent personally Involved In the selection and 
development process of the CHI? 

Y Yes 
N No 

U Unknown, uncertain 

50 What is respondent's attitude towards the CMI/updat1ng system? 
P Positive 

N Negative 

U Uncertain, unclear, undecided 
X Not nentloned, unknown 



51 - 52 Blank 



Sources of Resistance 



CODING MATRIX 



p. 8 



53 - 67 Put P as a personal concern of the respondent, 
Y as a concern of others, 
N 1n categories not reported. 

53 Too much and/or too rapid change 

54 Doubt about long term commitment to program 

55 Too late In year for Implementation 

56 Technical problems leading to frustration/avoidance 

Lack of support from: 

57 respected teachers 

58 teachers association 

59 principal 

60 district 

61 Processing/reporting too slow 

62 Problems with report or data (wrong names, misplaced Info) 

63 Fear of evaluation 

64 Inadequate training 

65 Too much paperwork 

66 Too much class time 

67 A lot/ too much time 

68 Coercion - mandated by district 

69 Lack of confidence In tests 

70 Teachers not asked for Input 

71 Coordination and supply of materials 

72 Other 

U l ; : ^ a? 

ERIC 



CODING MATRIX 

School Organization (Card 3) p. 9 

12 Collaboration with colleagues 
M Hore 

L Less 

N No Change 

U Uncertain 

X Unknown 

13 - 21 For these numbers use the appropriate letter Including: 

N No change 

X Unknown 

Changes In meetings 

13 A Yes, change In amount 

14 T Yes, change In type 

15 F Yes, change In frequency 

Change In school structure 

16 S New structure, e.g. lab 

17 G Grade level collaboration 

18 I Inter-grade level collaboration 

19 D More departmentalization 

20 C More self-contained classes 

21 0 Other 

22 Class schedule changes within the school 
Y Yes, there have been changes 

N No, there haven't been any changes 

X Unknown 

23 Staffing changes 

R Yes, changes In staffing responsibilities 

A Yes, adding staff 

D Yes, decreasing staff 

N No 

X Unknown 

24 Changes In teacher relations with students 
A Yes, positive 

B Yes, negative 

N No change 

X Unknown 

25 Changes In teacher relations with parents 
A Yes, positive 

B Yes, negative 

N No change 

X Unknown 



CODING MATRIX 

p. 10 

(C-V«'v ivi.i*"*'**}) 

26 - 27 Blank 
Benefits 

28-37 Possible benefits to using the system: code each of 
the following that apply; put 1n X when 1t doesn't. 

28 T Easier and/or faster to test 

29 U Easier to update 

30 D Standardization of district measures 

31 A Administrative needs 

32 I Instruction (planning, diagnostics, pacing, grouping) 

33 S Students 

34 P Parents 

35 R (makes teachers aware of) specific requirements and district 

expectations 

36 Y Accuracy of assessment/objectives measures 

37 F Improving Instruction by focusing on objectives or proficiencies 

38 -39 Blank 
Motivation/Incentives 

40 -44 Why do teachers Use updat1ng/CMI? 

40 P To please/appease the principal 

41 M Mandated 

42 C Curious/Interested 

43 B Potential benefits 

44 T Easier and/or faster 
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CODING MATRIX 
(Card 3, continued) 

45 - 46 Blank 



Procedural Aspects 

47 How much has the person used CHI? 
L A little 
S Some 
M A lot, much 
N Not at all 
X Unknown 

48 How has the person used CHI? 
L Uses tests without scanning 
S Uses tests with someone else scanning 
M Uses test and scans himself 

N Not at all 
X Unknown 

50 - 51 Blank 



Recommendations for future use 

52 C New, clearer test copies 

53 K One key person responsible for scanning on site 

54 T More technical training 

55 I Training for Instructional uses 

56 A More administrative support In building and district 

57 0 Other 
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